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Background/context: 

Description of prior research and/or its intellectual context and/or its policy context. 

This study aims to help address a need for information about efficacious mathematics 
curriculum and instruction at the high school level. Data from national and international studies, 
including the National Assessment of Educational Progress (NAEP), show very low mathematics 
proficiency rates for high school students. Many educators recognize a need to improve 
mathematics achievement through interventions such as new curriculum materials or 
instructional approaches, however they lack rigorous evidence on efficacy. By providing such 
information, this study can help guide educators in selecting the most promising materials. 

Purpose/objective/research question/focus of study: 

Description of what the research focused on and why. 

The purpose of the study is to evaluate the efficacy of the Cognitive Tutor Geometry 
curriculum by Carnegie Learning Inc., using randomized controlled trials. 

Setting: 

Specific description of where the research took place. 

Baltimore County Public Schools (BCPS) is in the urban fringe of Baltimore, serving 
students from a wide range of racial/ethnic and socioeconomic backgrounds. The district has 25 
high schools, with a total enrollment of about 32,000 students. 

Population/Participants/Subj ects : 

Description of participants in the study: who (or what) how many, key features (or characteristics). 

In BCPS, minority enrollment is 39% and low-SES enrollment is 17%. Eight schools are 
included in the study sample. In participating schools, students enrolled in the standard geometry 
classes meeting the study’s scheduling requirements were included in the random assignment 
process, regardless of grade level or other student characteristics. The study involved about 2,000 
students, although only a portion of the students met all eligibility requirements and completed 
both pre- and post-testing. 

Intervention/Program/Practice : 

Specific description of the intervention, including what it was, how it was administered, and its duration. 

The Cognitive Tutor Geometry curriculum was selected for study because it has shown 
promise in quasi-experimental studies; and because a related curriculum for Algebra I had 
significant positive effects in a randomized field trial (Morgan and Ritter, 2002). The curriculum 
is designed to promote understanding of geometric concepts and principles, and to enhance 
abstract and spatial reasoning skills. Students spend 40% of their class time using individualized 
tutorial software built on a detailed computational model of student thinking, and 60% in 
classroom activities that involve teacher-guided group work and problem solving. Teachers 
receive four days of training, a set of teacher training materials, and a teacher text. 
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Research Design: 

Description of research design (e.g., qualitative case study, quasi-experimental design, secondary analysis, analytic 
essay, randomized field trial). 

Two participating teachers in each school taught geometry concurrently during two distinct 
class periods. During the earlier class period, one was selected randomly to use the Cognitive 
Tutor Geometry curriculum and the other used the school’s existing geometry curriculum. 

During the later period, the teachers switched curricula. For both periods, students enrolled in 
geometry were randomly assigned to experimental or control classes. 

Data Collection and Analysis: 

Description of plan for collecting and analyzing data, including description of data. 

The main outcome measure was the district’s required final exam, which include both 
multiple-choice and extended-response items. The ETS end-of-course algebra exam was 
administered as a pretest. The study also gathered data on other student outcomes that may 
influence long-term student achievement, such as mathematics confidence and attitudes. To 
monitor implementation fidelity, the study conducted three site visits per year, observing each 
class, including computer lab classes. Control classrooms were also each observed three times. 
Observations followed a rubric modeled after guidelines the developer uses to support 
implementation, which cover important elements of the instructional design. During these visits, 
teachers were interviewed about implementation and instructional practices. 

Analyses used hierarchical linear models for test scores and generalized linear mixed models 
for dichotomous outcome measures, accounting for classroom period differences and teacher 
effects. Data from observation rubrics were synthesized into three scales of implementation for 
use in modeling the relationship between implementation and student outcomes. Qualitative 
analyses of site visit data also were used to interpret the observed treatment effects. 

Findings/Results: 

Description of main findings with specific details. 

The study found a significant negative effect of treatment on student achievement 
(standardized effect size of -0.25, p<0.05). This result is robust to alternative model 
specifications. Implementation data do not suggest that failure to implement the intervention 
curriculum explains the student achievement findings. The inquiry -based approach prescribed by 
the curriculum was unfamiliar to students and teachers, and our qualitative results suggest this 
may have impeded learning. The study design does not enable distinguishing the effects of the 
tutorial software from other aspects of the curriculum. 

Conclusions: 

Description of conclusions and recommendations of author(s) based on findings and over study. (To support the 
theme of 2009 conference, authors are asked to describe how their conclusions and recommendations might inform 
one or more of the above noted decisions — curriculum, teaching and teaching quality, school organization, and 
education policy.) 

The results of this study do not find positive effects of the Cognitive Tutor Geometry 
curriculum in this school context. It is not clear how much the challenge students and teachers 
faced in learning with an inquiry-based approach contributes to this negative finding. It is also 
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unclear whether those challenges are localized to the schools in the study. This could potentially 
limit the generalizability of these results. On the other hand, the results do contradict quasi- 
experiment studies that found large positive effects, and may suggest that randomized controlled 
trials can eliminate biases of other methods. 
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Appendixes 
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Appendix A. References 
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